Machine Learning Models Accurately Predict PTHrP Results But Fail To Predict Physician Behavior

Abstract Quantification of circulating parathyroid hormone-related peptide (PTHrP) aids in the diagnosis humoral hypercalcemia malignancy. However, this test is often ordered settings low pre-test probability or mistakenly when hormone (PTH) was desired. To improve utilization, all PTHrP orders at our institutions are reviewed by a laboratory medicine resident. If order appears to be inappropriate, ordering physician contacted ask for permission cancel. In work, we attempted automate labor-intensive review process developing machine learning models predict willing We collected 2171 that were subjected manual over past 10 years. removed any repeats, leaving 1649 first-time orders. For each order, assigned class label ‘canceled’ ‘completed’ based upon notes resident’s documentation logs. aggregated data patient existing within information system time first (n = 40 million). Various strategies applied impute missing data, including missingness as feature, fill medians, k-nearest neighbors, bagged trees, and linear regression. Class imbalances adjusted using synthetic minority upsampling technique (SMOTE) adaptive (ADASYN) final ratio one:one. The dataset partitioned into 70:30 split between training testing sets with five-fold cross-validation. Several algorithms trained, logistic regression, naive Bayes, random forest, XGBoost. After cross-validation, held-out set, performance evaluated area under receiver operating characteristic curve (AUC). XGBoost best performer predicting provider’s likelihood cancel test, but an AUC only 0.63. Surprised poor performance, devised second classification task results (normal vs abnormal, threshold 4.2 pmol/L) subset completed (n=1371). Using same pipeline described above, again observed performer, 0.89. striking difference trained on two different targets suggests physician’s willingness assent intervention may unrelated underlying biology. Likely explanations include reaching provider who not primary medical decision-maker correct wrong they too busy revisit details prior work. either case, reflex set defines specific criteria performing PHTrP upfront more effective way utilization.